Stochastic Games: A Tutorial

Game theory [1] is a formalism for the study of competitive interaction in the rich spectrum of relationships ranging between conflict and cooperation. Originally conceived as a mathematical foundation of economics, it proved its robustness by providing new techniques and insights in logic and set theory [15, 13], evolutionary and population biology [22], auction design and implementation, the design and study of the internet [19], analysis of cold war strategies, etc. Dynamic games are used to model competitive processes evolving over time. Stochastic transitions are used for abstraction in modelling or to formalize inherent uncertainty, leading to quantitative statistical analysis. Stochastic games are dynamic games with stochastic transitions. They enjoy rich and mature mathematical theory [11, 23] and a wide range of applications including economics, cell, population and evolutionary biology [26], queueing theory and performance evaluation, and quantitative temporal model checking [17, 8]. In this tutorial we explore the spectrum of competitive dynamic behavioural models ranging from Markov chains, to transition systems, to Markov decision processes [20], to 2-player and multi-player perfect-information and general stochastic games [11]. We also consider the spectrum of qualitative and quantitative objectives for the players contained in the class of Borel-measurable functions [16], and we survey most popular concrete choices: discounted and limiting-average objectives [20, 11] used in economics and performance evaluation, and omega-regular objectives [13] encompassing safety, liveness and fairness objectives used in temporal logic model checking. We present the general determinacy result for zero-sum stochastic games with Borel-measurable objectives due to Martin [16], as well as its many refinements for stochastic games with discounted, limiting-average, and omega-regular objectives [11, 6, 9, 17, 4, 28]. Then we survey algorithmic techniques for solving various classes of stochastic games and Markov decision processes, including value iteration and strategy improvement [20, 7], convex optimization [7, 27, 2] and graph-theoretic algorithms [3]. Special attention is given to algorithmic solutions of (perfect-information) omega-regular stochastic games [14, 3, 4] laying the foundation for automated quantitative temporal model checking. We also consider non-zero sum stochastic games. After defining the fundamental notion of a Nash equilibrium we survey recent results [24, 21, 5] on its existence in various classes of stochastic games as well as a few challenging open problems in the area. Then we present the quantitative mu-calculus [10, 17, 8] which is a very expressive, robust, and yet algorithmically feasible logical formalism for specification and automated quantitative analysis. We argue that the quantitative